Deep learning approach for assessing credit risk

ABSTRACT

Systems and methods to facilitate credit risk assessment are described herein. The systems and methods described herein relate to implementing and training a credit risk model comprising a document model and a company model. The document model may be configured to read text of a document, understand long range relationships between words, phrases, and the occurrence of one or more financial events, and create a document score that indicates whether the financial events are likely to occur based on that document. A document-model-state vector may be generated that represents important features and relationships identified within each document and across a set of documents for a given entity based on the document scores. The company model may produce a sequence of default probability scores representing overall likelihoods of the occurrence of the financial events for an entity based on the document-model-state vector for documents associated with that entity.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication Ser. No. 62/655,974, entitled “DEEP LEARNING APPROACH FORASSESSING CORPORATE CREDIT RISK”, filed Apr. 11, 2018, which is herebyincorporated by reference in its entirety.

FIELD OF THE INVENTION

The systems and methods described herein relate to assessing creditrisk.

BACKGROUND

Filing for bankruptcy and/or defaulting on a debt can adversely affectthe credit of an entity, such as a company or firm. It is important tobe able to identify adverse future events an entity may face andestablish procedures to measure, mitigate, and manage risk. For example,investors extending credit to entities must be able to assess potentiallosses resulting from such activity. Similarly, investors who hold anequity interest in entities must be able to assess potential volatilityaffecting those investments and adjust their portfolios accordingly.

Various credit risk models exist to assess the default probability of anentity. A broad range of sources may be utilized by these credit riskmodels. Many of these data sources may be derived directly from publicdata sources. For example, various existing credit risk models may rateentities (such as firms or companies) based on likelihood of defaultingon their debt using financial accounting data, such as accounting ratiosand pricing information from pricing services. Example metrics computedin existing credit risk models may include a probability of default(e.g., the likelihood that an entity will fail to meet its financialobligations) and loss given default (e.g., if a default occurs, theamount those who extended credit to the entity expect to lose).

While the sources of information utilized by existing credit risk modelsprovide valuable input to the credit risk modeling process, there is avast amount of publicly available information that is overlooked byexisting models. For example, textual based data sources such as newsarticles that report on a firm's past, current, and possible futureevents typically include important information that is not considered inthe credit risk modeling process. Further, the semantic context of textincluded in these data sources is typically not analyzed by theseprocesses. Accordingly, there is a need for an improved credit riskmodeling process capable of considering documents related to an entity,including the unstructured text in these documents, in addition to thefinancial account data considered by currently existing credit riskmodels.

SUMMARY

The systems and methods described herein relate to implementing andtraining a credit risk model. For example, the systems and methodsdescribed herein may relate to a system configured to utilize anext-generation deep learning neural network to computationally assesscredit quality for various entities. In various implementations, thesystem may analyze unstructured text from documents including one ormore of news, research, filings, transcripts, and/or other textual ortabular sources to determine relationships between words and phrasesthat are indicative of one or more future financial events (e.g.,financial deterioration such as bankruptcy or default, or improvement).The system may understand relationships in the meaning and/or context ofwords and phrases in the individual documents without preprocessing thetext or creating stop words, n-grams, and/or other dictionaries used fortext mining in a bag-of-words model.

The credit risk model utilized by the systems and methods describedherein represents an improvement over existing credit risk models. Forexample, by unifying several different credit risk modeling approacheswithin a single model framework, the credit risk model described hereinproduces an improved assessment of the default probability of an entitythat takes into account additional information available related to anentity, such as documents related to that entity comprising unstructuredtext. The improved credit risk model is able to look at these documentsboth individually and collectively to identify relationships across acollection of documents. Additionally, the credit risk model describedherein produces a sequence of default probability scores representingoverall likelihoods of the occurrence of one or more financial eventsfor an entity. Accordingly, unlike conventional credit risk models, thesystems and methods described herein are able to produce an indicationof the default probability of an entity at various intervals over aperiod of time, rather than a single assessment of the defaultprobability of an entity at one particular time.

In various implementations, the systems and methods described herein mayutilize a deep learning neutral network comprising a document model anda company model. The document model may use a deep-learning model withmemory to read text of a document, understand long range relationshipsbetween words, phrases, and the occurrence of the one or more financialevents, and create a document score that indicates whether the one ormore financial events are likely according to the document. The documentmodel may generate a document-model-state vector that representsimportant features and relationships identified within each document andacross a set of documents for a given entity. The company model mayproduce a sequence of default probability scores representing overalllikelihoods of the occurrence of one or more financial events for anentity based on the document-model-state vector for the documentsassociated with that entity. The company model may aggregate thedocument-model-state vector with financial information for the entity toproduce individual default probability scores in the sequence of defaultprobability scores. The financial information may include one or more ofpricing data, fundamental ratios, and/or other tabular data.

The system described herein may include one or more of servers, clientcomputing platforms, document sources, financial sources, externalresources, and/or other components. In various implementations, thesystem may include one or more servers comprising electronic storage andone or more physical processors. The one or more physical processors maybe configured by computer-readable instructions. Executing thecomputer-readable instructions may cause the one or more physicalprocessors to implement and train a credit risk model. Thecomputer-readable instructions may include one or more computer programcomponents. The computer program components may include one or more of adocument model component, a company model component, a model trainingcomponent, and/or other computer program components. The one or morephysical processors may represent processing functionality of multiplecomponents of the system operating in coordination. Therefore, thevarious processing functionality described in relation to the one ormore processors may be performed by a single component or by multiplecomponents of the system.

The document model component may be configured to implement a documentmodel to produce individual document scores for individual documentsrepresenting a likelihood of the occurrence of one or more financialevents for an entity based on those individual documents. In variousimplementations, the document model component operates as an encoder togenerate a “document-representation”—or document-representationvector—for each document associated with a given entity and generated,obtained, published, and/or otherwise made available during a period oftime. For example, the document model component may be configured togenerate a document-representation vector for each document in atrailing history (e.g., the last 12 months). In various implementations,the document score produced for a document is based on the generatedvector(s) for that document. In various implementations, individualdocument scores may be individual predictive descriptors comprising anumber between zero (0) and one (1), with zero (0) indicating a defaultand/or bankruptcy being very unlikely and one (1) indicating a defaultand/or bankruptcy being likely. In various implementations, the documentmodel component aggregates the document-representation vectors anddocument scores to create a document-model-state vector. Thedocument-model-state vector represents important features andrelationships identified within each document and across a set ofdocuments for a given entity.

The company model component may be configured to implement a companymodel to produce a sequence of default probability scores representingoverall likelihoods of the occurrence of one or more financial eventsfor an entity based on the document-model-state vector for the documentsassociated with that entity. In various implementations, the companymodel component operates as a decoder to output company scores (e.g.,between zero (0) and one (1)). A company score may represent a defaultprobability for an entity. Based on the document-model-state vectorcreated by the document model component and other inputs (such as entityfinancial information), the company model component may produce companyscores along with an updated internal state vector—thecompany-model-state vector. The company model may comprise an adaptationof one or more deep learning models typically used for text translation,including a sequence-to-sequence model. Instead of training the model totranslate text (e.g. a sequence of English words to a sequence of Frenchwords) the company model may be trained to translate a sequence ofclassifier states and document scores to the sequence of defaultprobability scores.

The model training component may be configured to utilize abackpropagation algorithm to train the deep learning neutral network. Invarious implementations, the model training component may be configuredto define each layer of the document model and/or company model (i.e.,the credit risk model) and initialize each parameter with random values.In various implementations, the model training component may beconfigured to run the full model with thousands or millions ofhistorical examples to determine whether the output from the full modelmatches the desired output. In some implementations, the model trainingcomponent may be configured to train the document model and the companymodel individually. In some implementations, the model trainingcomponent may be configured to train the document model and the companymodel in a single training step, rather than training each modelindividually.

These and other objects, features, and characteristics of the systemand/or method disclosed herein, as well as the methods of operation andfunctions of the related elements of structure and the combinationthereof, will become more apparent upon consideration of the followingdescription and the appended claims with reference to the accompanyingdrawings, all of which form a part of this specification, wherein likereference numerals designate corresponding parts in the various figures.It is to be expressly understood, however, that the drawings are for thepurpose of illustration and description only and are not intended as adefinition of the limits of the invention. As used in the specificationand in the claims, the singular form of “a”, “an”, and “the” includeplural referents unless the context clearly dictates otherwise.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings are provided for purposes of illustration only and merelydepict typical or example implementations. These drawings are providedto facilitate the reader's understanding and shall not be consideredlimiting of the breadth, scope, or applicability of the disclosure. Forclarity and ease of illustration, these drawings are not necessarilydrawn to scale.

FIG. 1 illustrates an example of a system configured to implement andtrain a credit risk model for assessing the credit quality of an entity,in accordance with one or more implementations.

FIG. 2 illustrates an example of a document model of a credit riskmodel, in accordance with one or more implementations.

FIG. 3 illustrates an example of a company model of a credit risk model,in accordance with one or more implementations.

FIG. 4 illustrates an example of a method for assessing the creditquality of an entity utilizing a document model and a company model, inaccordance with one or more implementations.

DETAILED DESCRIPTION

FIG. 1 illustrates a system 100 configured to implement and train acredit risk model for assessing the credit quality of an entity, inaccordance with one or more implementations. In various implementations,system 100 may be configured to utilize a deep learning, convolutionalneural network comprising a document model and a company model. The deeplearning neural network may be configured to computationally assesscredit quality for various entities. For example, the deep learningneural network may be configured to predict a number of future financialevents including, but not limited to, probability of default orbankruptcy, loss given default, probability of rating agency ratingchange, probability of equity price moves, and/or other financialevents.

In various implementations, system 100 may include one or more of one ormore servers 102, one or more client computing platforms 104, one ormore document sources 106, one or more financial sources 108, externalresource(s) 110, electronic storage 140, and/or other components. Insome implementations, one or more servers 102 and one or more othercomponents of system 100 may be included within a single deviceconfigured to implement and train a credit risk model. For example, oneor more servers 102 and electronic storage 140 may be included within asingle device.

In some implementations, server(s) 102, one or more client computingplatforms 104, one or more document sources 106, one or more financialsources 108, external resources 110, and/or one or more other componentsof system 100 may be operatively linked via one or more electroniccommunication links. For example, such electronic communication linksmay be established, at least in part, via a network 130 such as theInternet, Bluetooth, and/or other networks. It will be appreciated thatthis is not intended to be limiting, and that the scope of thisdisclosure includes implementations in which server(s) 102, one or moreclient computing platforms 104, one or more document sources 106, one ormore financial sources 108, and/or other components may be operativelylinked via some other communication media.

A given client computing platform 104 may include one or more processorsconfigured to execute computer program components. The computer programcomponents may be configured to enable an expert or user associated withthe given client computing platform 104 to interface with system 100and/or external resources 110, and/or provide other functionalityattributed herein to client computing platform(s) 104. By way ofnon-limiting example, a given client computing platform may include oneor more of a desktop computer, a laptop computer, a handheld computer, atablet computing platform, a NetBook, a Smartphone, and/or othercomputing platforms.

External resources 110 may include sources of information outside ofsystem 100, external entities participating with system 100, and/orother resources. In some implementations, some or all of thefunctionality attributed herein to external resources 110 may beprovided by resources included in system 100.

Illustration of one or more processors 112 in FIG. 1 is not intended tobe limiting. The one or more processors 112 may include a plurality ofhardware, software, and/or firmware components operating together toprovide the functionality attributed herein to one or more processors112. For example, one or more processors 112 may be implemented by acloud of computing platforms operating together as one or moreprocessors 112.

Electronic storage 140 may comprise non-transitory storage media thatelectronically stores information. The electronic storage media ofelectronic storage 140 may be provided integrally (i.e., substantiallynon-removable) with one or more components of system 100 and/orremovable storage that is connectable to one or more components ofsystem 100 via, for example, a port (e.g., a USB port, a Firewire port,etc.) or a drive (e.g., a disk drive, etc.). Electronic storage 140 mayinclude one or more of optically readable storage media (e.g., opticaldisks, etc.), magnetically readable storage media (e.g., magnetic tape,magnetic hard drive, floppy drive, etc.), electrical charge-basedstorage media (e.g., EPROM, EEPROM, RAM, etc.), solid-state storagemedia (e.g., flash drive, etc.), and/or other electronically readablestorage media. Electronic storage 140 may include one or more virtualstorage resources (e.g., cloud storage, a virtual private network,and/or other virtual storage resources). Although electronic storage 140is shown in FIG. 1 as a single entity, this is for illustrative purposesonly. In some implementations, electronic storage 140 may comprisemultiple storage units. These storage units may be physically locatedwithin the same device, or electronic storage 140 may represent storagefunctionality of multiple devices operating in coordination.

Electronic storage 140 may store software algorithms, informationdetermined by processor(s) 112, information received from server(s) 102,information received from one or more client computing platforms 104,information received from one or more document sources 106, informationreceived from one or more financial sources 108, and/or otherinformation that enables server(s) 102 to function as described herein.

In various implementations, server(s) 102 may further includenon-transitory electronic storage. For example, server(s) 102 mayinclude non-transitory electronic storage the same as or similar toelectronic storage 140. In various implementations, non-transitoryelectronic storage of server(s) 102 may be configured to store amachine-learning algorithm and/or other information configured tofacilitate cloud-based point-to-point data transfer. Themachine-learning algorithm may include an unsupervised goal-basedmachine learning program. The machine-learning algorithm may be providedinput from one or more components of computer readable instructions 114.Compared to supervised learning where training data is labeled with theappropriate classifications, unsupervised learning may learnrelationships between elements in the provided data sets and provideoutput without user input. The relationships can take many differentalgorithmic forms, but some models may have the same goal of mimickinghuman logic by searching for indirect hidden structures, patterns,and/or features to analyze new data.

Implementations of the disclosure may be made in hardware, firmware,software, or any suitable combination thereof. Aspects of the disclosuremay be implemented as instructions stored on a machine-readable medium,which may be read and executed by one or more processors. Amachine-readable medium may include any mechanism for storing ortransmitting information in a form readable by a machine (e.g., acomputing device). For example, a tangible computer readable storagemedium may include read only memory, random access memory, magnetic diskstorage media, optical storage media, flash memory devices, and others,and a machine-readable transmission media may include forms ofpropagated signals, such as carrier waves, infrared signals, digitalsignals, and others. Firmware, software, routines, or instructions maybe described herein in terms of specific exemplary aspects andimplementations of the disclosure, and performing certain actions.

Although processor(s) 112 is illustrated in FIG. 1 as a singlecomponent, this is for illustrative purposes only. In someimplementations, processor(s) 112 may comprise multiple processingunits. These processing units may be physically located within the samedevice, or processor(s) 112 may represent processing functionality ofmultiple devices operating in coordination. Processor(s) 112 may beconfigured to execute one or more components by software; hardware;firmware; some combination of software, hardware, and/or firmware;and/or other mechanisms for configuring processing capabilities onprocessor(s) 112. As used herein, the term “component” may refer to anycomponent or set of components that perform the functionality attributedto the component. This may include one or more physical processorsduring execution of processor readable instructions, the processorreadable instructions, circuitry, hardware, storage media, or any othercomponents. Furthermore, it should be appreciated that although thevarious instructions are illustrated in FIG. 1 as being co-locatedwithin a single processing unit, in implementations in whichprocessor(s) 110 include multiple processing units, one or moreinstructions may be executed remotely from the other instructions.

The description of the functionality provided by the differentcomputer-readable instructions described herein is for illustrativepurposes, and is not intended to be limiting, as any of instructions mayprovide more or less functionality than is described. For example, oneor more of the instructions may be eliminated, and some or all of itsfunctionality may be provided by other ones of the instructions. Asanother example, processor(s) 110 may be programmed by one or moreadditional instructions that may perform some or all of thefunctionality attributed herein to one of the computer-readableinstructions.

In various implementations, one or more servers 102 (alsointerchangeably referred to herein as server(s) 102, server 102, orservers 102 for convenience) may be configured to communicate with oneor more client computing platforms 104, one or more document sources106, one or more financial sources 108, external resource(s) 110, and/orother components of system 100 according to a client/serverarchitecture, peer-to-peer architecture, and/or other architectures.Users may access system 100 via client computing platform(s) 104. Theserver(s) 102 may provide the client computing platform(s) 104 withinformation necessary to present user interfaces on the client computingplatform(s) 104. The client computing platform(s) 104 may communicateinformation back to server(s) 102 in order to facilitate furtherprocessing and/or functionality at server(s) 102. Communications may befacilitated through network(s) 130. The network(s) 130 may include wiredand/or wireless connections. The network(s) 130 may include theInternet, Bluetooth, USB, and/or other communication networks. It willbe appreciated that this is not intended to be limiting and that thescope of this disclosure includes implementations in which components ofsystem 100 may be operatively linked via some other communication media.

Individual document sources of one or more document sources 106 mayinclude entities which publish and/or otherwise make publicly availabledocuments about one or more entities. The documents may comprise one ormore of news, research, filings, transcripts, and/or other textual ortabular sources that include unstructured text. In variousimplementations, the systems and methods described herein may beconfigured to analyze the unstructured text in the documents todetermine relationships between words and phrases that are indicative ofone or more future financial events (e.g., financial deterioration suchas bankruptcy or default, or improvement). By way of non-limitingillustration, an individual document source may include one or more of abusiness entity, a reporting agency, and/or other source. The documentsmay include content indicative of one or more future financial events(e.g., financial deterioration such as bankruptcy or default, orimprovement) for one or more entities. In some implementations, thedocuments may include web document accessed from webpages associatedwith the individual document sources.

Individual financial sources of one or more financial sources 108 mayinclude entities which publish and/or otherwise make publicly availablefinancial information for one or more entities. The financialinformation may include one or more of pricing data, fundamental ratios,and/or other information. By way of non-limiting illustration, anindividual financial source may include one or more of a businessentity, a reporting agency, and/or other source.

In various implementations, server(s) 102 may include one or morephysical processors 112 and/or other components. The one or morephysical processors 112 (also interchangeably referred to herein asprocessor(s) 112, processor 112, or processors 112 for convenience) maybe configured to provide information processing capabilities in system100. As such, the processor(s) 112 may comprise one or more of a digitalprocessor, an analog processor, a digital circuit designed to processinformation, a central processing unit, a graphics processing unit, amicrocontroller, an analog circuit designed to process information, astate machine, and/or other mechanisms for electronically processinginformation.

In various implementations, processor(s) 112 may be configured toexecute one or more computer readable instructions 114. Executing thecomputer readable instructions 114 may cause one or more processors 112to implement and train a credit risk model. Computer readableinstructions 114 may include one or more computer program components. Insome implementations, computer readable instructions 114 may be storedin electronic storage (e.g., electronic storage 140). Computer readableinstructions 114 may include one or more of document model component116, company model component 118, model training component 120, and/orother computer program components. As used herein, for convenience, thevarious computer readable instructions 114 will be described asperforming an operation, when, in fact, the various instructions programthe processor(s) 112 (and therefore system 100) to perform theoperation.

Document model component 116 may be configured to utilize a documentmodel to produce individual document scores for individual documentsrepresenting a likelihood of the occurrence of one or more financialevents for an entity based on the individual documents. The documentmodel may use a deep-learning model with memory to read text of adocument, understand long range relationships between words, phrases,and the occurrence of the one or more financial events, and create adocument score that indicates whether the one or more financial eventsare likely according to the document. The document model may understandrelationships in the meaning and/or context of words and phrases in theindividual documents without preprocessing the text or creating stopwords, n-grams, and/or other dictionaries used for text mining in abag-of-words model. The memory of the document model may represent aclassifier state at a given period in time and/or over a certain timeperiod.

In various implementations, document model component 116 may beconfigured to obtain individual documents related to at least oneentity. For example, document model component 116 may be configured toobtain individual documents from one or more document sources 106. Insome implementations, document model component 116 may be configured toobtain individual documents from one or more document sources 106sequentially as they are made accessible (e.g., published) over time.

In various implementations, document model component 116 may beconfigured to analyze unstructured text from documents including one ormore of news, research, filings, transcripts, and/or other textual ortabular sources to determine relationships between words and phrasesthat are indicative of one or more future financial events (e.g.,financial deterioration such as bankruptcy or default, or improvement).For example, document model component 116 may be configured to analyzeunstructured text from individual documents obtained from one or moredocument sources 106.

In various implementations, document model component 116 may beconfigured to generate a “document-representation”—ordocument-representation vector—for each document associated with a givenentity. For example, document model component 116 may be configured togenerate a document-representation vector for each document associatedwith a given entity that is generated, obtained, published, and/orotherwise made available during a period of time. In someimplementations, document model component 116 may be configured togenerate a document-representation vector for each document in atrailing history (e.g., the last 12 months). In various implementations,document model component 116 may be configured to generate adocument-representation vector for individual documents obtained fromone or more document sources 106.

In various implementations, document model component 116 may beconfigured to obtain, via the document model, individual document scoresfor individual documents associated with individual entities. Forexample, an entity may comprise a company, firm, and/or other businessentity. In various implementations, document model component 116 may beconfigured to produce, for each of a set of documents, a document scorethat indicates whether the one or more financial events are likely tooccur based on each individual document. In various implementations,document model component 116 may be configured to produce a documentscore for a document based on the document-representation vector(s)generated for that document.

In various implementations, document model component 116 may beconfigured to obtain and/or arrange a set of document scores as asequence of document scores. The sequence may represent a timeline overwhich the individual documents were made available and individual scoresassigned. In some implementations, document model component 116 may beconfigured to obtain a group of documents over a certain time period(e.g., a day, week, month, etc.) to produce one or more documents scoresfor that time period. In some implementations, document model component116 may be configured to consecutively obtain groups of documents overconsecutive time periods (e.g., a day, week, month, etc.) to produce agraph of documents scores per time period as a function of time. In someimplementations, document model component 116 may be configured toaggregate multiple documents together and process them as a singledocument. For example, document model component 116 may process multipledocuments relating to an entity over a certain time period (e.g., a day,week, month, etc.) as one document. As used herein the word “document”may include a single document, a group of multiple documents, a portionof a single document, and/or a combination thereof.

Individual document scores may represent a likelihood of the occurrenceof one or more future financial events for the entity based on theindividual documents. In various implementations, document modelcomponent 116 may be configured to determine a document score for anindividual document based on the relationships between words and/orphrases in that document and the likelihood of the occurrence of one ormore financial events. In other words, the appearance of certain wordsand/or phrases in a document indicate a higher (or lower) likelihood ofthe occurrence of one or more financial events. In variousimplementations, document model component 116 may be configured todetermine a document score for a document via a deep learning neuralnetwork and/or other techniques. In some implementations, the individualdocument scores may be individual predictive descriptors comprising anumber between zero (0) and one (1), with zero (0) indicating a defaultand/or bankruptcy being very unlikely and one (1) indicating a defaultand/or bankruptcy being likely. In some implementations, document modelcomponent 116 may be configured to distinguish between documentsconveying a likelihood of the occurrence of a future financial event anddocuments which merely include negative information. Negativeinformation may refer to information that may be adverse to a reputationof an entity but not necessarily indicative of default, bankruptcy,and/or other financial event. In various implementations, document modelcomponent 116 may be configured to aggregate the document-representationvectors and document scores to create a document-model-state vector. Adocument-model-state vector represents features and relationshipsidentified within each document and across a set of documents for agiven entity.

In various implementations, the document scores may be the same as, orproduced in a similar manner to, predictive descriptors, labels, featurevectors, and/or other information, as described in U.S. Pat. No.8,671,040 to Ryan D. Roser et. al., filed Jul. 23, 2010, and entitled“CREDIT RISK MINING,” the disclosure of which is hereby incorporated byreference in its entirety herein. By way of non-limiting illustration,the document model utilized by document model component 110 may be thesame as, or similar to, one or more models described in U.S. Pat. No.8,671,040.

FIG. 2 illustrates an example of a document model 200 of a credit riskmodel, in accordance with one or more implementations. In variousimplementations, document model component 116 may be configured toutilize a document model the same as or similar to document model 200 toproduce individual document scores for individual documents representinga likelihood of the occurrence of one or more financial events for anentity based on the individual documents. In various implementations,document model 200 may comprise an input layer 204 configured to receivedocuments 202. For example, documents 202 may comprise one or moredocuments received from one or more document sources 106. For eachdocument received, document model 200 may be configured to createdocument vectors. In various implementations, the document vectorscreated based on documents 202 are passed through an embedding layer 206that is configured to create a word embedding representation of thefeatures in that document (i.e., a document-representation vector). Thatvector is aggregated using several layers in a neural network (e.g., oneor more layers of document model 200 and company model 300, as describedfurther herein). A convolutional network may be passed over the vectorsto get a representation of that document that is derived from and hasextracted information indicating longer term relationships across thedocuments. For example, the word embedding representations created foreach of documents 202 are then aggregated via batch normalization 208.In various implementations, a GRU network (i.e., GRU layer 210) isconfigured to take the sequence of documents passed in and generate anoutput comprising a document-model-state vector 212. The GRU network maycomprise a sequential network through which the aggregated documentvectors are passed, and the state of the network is updated as eachdocument vector is read in order. The state of that network issequentially updated, and once each document vector has been passedthrough the GRU network, the final state of that network is representedby the document-model-state vector 212 that is based on the collectionof documents (i.e., documents 202). The document-model-state vector 212is then passed to company model 300.

Returning back to FIG. 1, document model component 116 may be configuredto cause a document-model-state vector for a given entity to be providedto company model component 118. In other words, based on the inputreceived (e.g., individual documents from one or more document sources106), document model component 118 may be configured to act as anencoder, aggregating generated document-representation vectors anddocument scores to create a document-model-state vector, which is outputto the company model component 118.

Company model component 118 may be configured to utilize a company modelto produce a sequence of default probability scores representing overalllikelihoods of the occurrence of one or more financial events for anentity. For example, company model component 118 may be configured toproduce a sequence of default probability scores representing overalllikelihoods of the occurrence of one or more financial events for anentity based on aggregated document-representation vectors and documentscores (i.e., the document-model-state vector) for documents associatedwith that entity. In various implementations, company model component118 may be configured to operate as a decoder to output a company score(e.g., between zero (0) and one (1)) for a given entity. The companyscore may represent a default probability for that entity (e.g. anindividual default probability score). An individual default probabilityscore may represent an individual overall (e.g., wholistic) likelihoodof occurrence of one or more future financial events based on thedocument-model-state vector for the documents associated with thatentity, financial information, and/or other information which may havebeen concurrently obtained within an individual certain period of time.Accordingly, a sequence of default probability scores may represent atimeline of overall likelihoods of occurrence of one or more futurefinancial events for a given entity.

In various implementations, company model component 118 may beconfigured to obtain a set of financial information for individualentities. For example, company model component 118 may be configured toobtain a set of financial information from one or more financial sources108. The financial information may include one or more of pricing data,fundamental ratios, and/or other tabular data.

In various implementations, company model component 118 may beconfigured to generate, via the company model, a sequence of defaultprobability scores representing the timeline of overall likelihoods ofoccurrence of one or more future financial events for the entity. Forexample, company model component 118 may be configured to produce acompany score for an entity based on a document-model-state vectorcreated by document model component 116. In various implementations,company model component 118 may be configured to aggregate adocument-model-state vector and/or other information obtained over acertain time period with financial information also obtained within thattime period for the entity to produce individual default probabilityscores within the sequence of default probability scores. The financialinformation may be used by the company model to help calibrate the stateof the model and provide context for the state vector. For example, ameasurement of a company's trailing volatility may be input along withat least the document-model-state vector, resulting in the production ofa less volatile sequence of company scores by company model component118 for entities with lower trailing volatility, whereas entities withhigher volatility would see a larger month-to-month movement in theirsequence of company scores.

Based on the document-model-state vector created by the document modelcomponent and other inputs (such as entity financial information), thecompany model component 118 may be configured to produce company scoresalong with an internal state vector—the company-model-state vector. Thecompany-model-state vector may be same shape as the document-model-statevector. In various implementations, company model component 118 may beconfigured to, on second and subsequent iterations, input acompany-model-state vector from a prior iteration to produce an updatedcompany score and updated company-model-state vector. For example,company model component 118 may be configured to input acompany-model-state vector from a prior iteration and one or more othertabular inputs (e.g., entity financial information) to again output acompany score (i.e., an updated company score) and an updatedcompany-model-state vector. In an example implementation, the companymodel utilized by company model component 118 may be trained such thateach iteration corresponds to one month, and company model component 118is configured to run twelve (12) iterations to calculate a 12-monthcurve of company scores (or default probability predictions).

In some implementations, the company model utilized by company modelcomponent 118 may comprise an adaptation of one or more deep learningmodels typically used for text translation. A deep learning model usedfor text translation may include a sequence-to-sequence model and/orother model. Instead of training the company model to translate text(e.g. a sequence of English words to a sequence of French words,) thecompany model may be trained to translate a sequence of classifierstates and/or document scores (e.g., obtained from the document model ofdocument model component 110) to the sequence of default probabilityscores.

FIG. 3 illustrates an example of a company model 300 of a credit riskmodel, in accordance with one or more implementations. In variousimplementations, company model component 118 may be configured toutilize a company model the same as or similar to company model 300 toproduce a sequence of default probability scores representing overalllikelihoods of the occurrence of one or more financial events for anentity. In various implementations, company model 300 may receive asinput financial information 302 and a document-model-state vector (e.g.,document-model-state vector 212) generated by a document model (e.g.,document model 200). In various implementations, company model 300 maycomprise an input layer 304 configured to receive financial information302. For example, financial information 302 may comprise financialinformation received from one or more financial sources 108. In variousimplementations, company model 300 may include a dense layer 306 and GRUlayer 308 configured to produce company scores and a company-model-statevector. Dense layer 306 may comprise a regular layer of neurons in aneural network, wherein each neuron receives input from all the neuronsin the previously layer—thus, the layer is “densely” connected. Invarious implementations, dense layer 306 may be configured to perform alinear operation on the layer's input vector. GRU layer 308 may comprisea layer the same as or similar to GRU layer 210, described herein withrespect to FIG. 2. In various implementations, the output of GRU layer308 is then aggregated via batch normalization 310. In variousimplementations, company model 300 may comprise an additional denselayer the same as or similar to dense layer 306 (i.e., dense layer 312).In various implementations, company model 300 may be configured tooutput at least a sequence of default probabilities (i.e., sequence ofdefault probabilities 314) representing overall likelihoods of theoccurrence of one or more financial events for an entity to which theone or more documents (i.e., documents 202) relate.

Returning back to FIG. 1, model training component 120 may be configuredto utilize a backpropagation algorithm to train the deep learningneutral network described herein that comprises both a document modeland a company model. In various implementations, model trainingcomponent 120 may be configured to define each layer of the documentmodel and company model (hereinafter referred to as the “full model”)and initialize each parameter with random values. In variousimplementations, model training component 120 may be configured to runthe full model with thousands or millions of historical examples todetermine whether the output from the full model matches the desiredoutput. In some implementations, model training component 120 may beconfigured to train the document model and the company modelindividually. In some implementations, model training component 120 maybe configured to train the document model and the company model in asingle training step (i.e., as a single “full model”). The single “fullmodel” may also be referred to herein as a credit risk model comprisingboth a document model and company model.

In some implementations, model training component 120 may be configuredto use historical entity defaults and bankruptcies to train the documentmodel and/or the company model. For example, model training component120 may be configured to train the document model and/or the companymodel based on a specific historical entity event. In an exemplaryimplementation, model training component 120 may be configured to obtaindocuments generated, obtained, published, and/or otherwise madeavailable during a 12-month time period prior to the event, along withfinancial information for that entity known at that time. These inputsare used by the document model and the company model (i.e., documentmodel component 116 and company model component 118, respectively) tocalculate the sequence of default probability scores (or companyscores). Model training component 120 may be configured to compare thissequence to historical bankruptcy and default records. With the date onwhich the specific historical entity event occurred known, the companyscore for the iteration pertaining to that date should be a one(1)—indicating a 100% likelihood of that historical financial eventoccurring. Model training component 120 may be configured to calculatethe difference between that company score and the desired value (1) tomeasure the error in the score. Using a backpropagation algorithm, modeltraining component 120 may be configured to use the measured error tomake tiny adjustments to the parameters in the layers of the model(s).In various implementations, model training component 120 may beconfigured to repeat this process across multiple examples to train themodel.

In some implementations, model training component 120 may be configuredto train the document model and/or the company model using one or moreother training techniques. For example, in some implementations, modeltraining component 120 may be configured to train the document modeland/or the company model using one or more training techniques describedin U.S. Pat. No. 8,671,040 to Ryan D. Roser et. al., filed Jul. 23,2010, and entitled “CREDIT RISK MINING,” the disclosure of which ishereby incorporated by reference in its entirety herein.

Exemplary Flowcharts of Processes

FIG. 4 illustrates a method 400 for assessing the credit quality of anentity utilizing a document model and a company model, in accordancewith one or more implementations. The operations of method 400 presentedbelow are intended to be illustrative and, as such, should not be viewedas limiting. In some implementations, method 400 may be accomplishedwith one or more additional operations not described, and/or without oneor more of the operations discussed. In some implementations, two ormore of the operations may occur substantially simultaneously. Thedescribed operations may be accomplished using some or all of the systemcomponents described in detail above.

In some implementations, method 400 may be implemented in one or moreprocessing devices (e.g., a digital processor, an analog processor, adigital circuit designed to process information, a central processingunit, a graphics processing unit, a microcontroller, an analog circuitdesigned to process information, a state machine, and/or othermechanisms for electronically processing information). The one or moreprocessing devices may include one or more devices executing some or allof the operations of method 400 in response to instructions storedelectronically on one or more electronic storage mediums. The one ormore processing devices may include one or more devices configuredthrough hardware, firmware, and/or software to be specifically designedfor execution of one or more of the operations of method 400.

In an operation 402, method 400 may include generating document scoresand document-representation vectors for each of a set of documentsassociated with an entity. In various implementations, a set ofdocuments associated with an entity may be obtained from one or moredocument sources. The documents may comprise documents associated with agiven entity and generated, obtained, published, and/or otherwise madeavailable during a predefined time period. In various implementations,the document scores and document-representation vectors may be generatedbased on the text of the documents utilizing a document model. Thedocument score for a given document indicates a likelihood of theoccurrence of one or more future financial events for a given entitybased on that document. The future financial event may comprise defaultand/or bankruptcy. In some implementations, operation 402 may beperformed by a processor component the same as or similar to documentmodel component 116 (shown in FIG. 1 and described herein).

In an operation 404, method 400 may include aggregating the documentscores and document-representation vectors. In some implementations,operation 404 may be performed by a processor component the same as orsimilar to document model component 116 (shown in FIG. 1 and describedherein).

In an operation 406, method 400 may include creating adocument-model-state vector based on the aggregated document scores anddocument-representation vectors. The document-model-state vector mayrepresent relationships identified within each of the documents to whichthe document scores and document-representation vectors are associatedand relationships identified across the documents. In variousimplementations, the document-model-state vector may comprise the outputof a document model. In some implementations, operation 406 may beperformed by a processor component the same as or similar to documentmodel component 116 (shown in FIG. 1 and described herein).

In an operation 408, method 400 may include producing a sequence ofdefault probability scores representing the likelihood of at least onefinancial event occurring based on the document-model-state vector. Thesequence of default probability scores may be produced utilizing aneural network comprising a document model and a company model. Invarious implementations, the document-model-state vector output from thedocument model and financial information for the entity may beaggregated. The sequence of default probability scores may be determinedbased on the aggregated document-model-state vector and financialinformation. In various implementations, the sequence of defaultprobability scores may comprise the output of the company model. In someimplementations, a company score may be generated based on the sequenceof default probability scores. The company score may comprise a singlevalue between zero (0) and one (1) and represent a default probabilityfor the entity. In some implementations, an internal state vector—acompany-model-state vector—may be generated based on thedocument-model-state vector utilizing the company model. In someimplementations, additional iterations may be run to update thecompany-model-state vector. For example, the company model may receiveas input a company-model-state vector generated in a prior iterationbased on a first document-model-state vector along with a seconddocument-model-state vector that was generated based on aggregateddocument scores and document-representation vectors for a second set ofdocuments. Utilizing the company model, a second sequence of defaultprobability scores may be produced based at least on the seconddocument-model-state vector, and an updated company-model-state vectormay be generated based on the company-model-state vector generated inthe prior iteration and the second document-model-state vector. In someimplementations, operation 408 may be performed by a processor componentthe same as or similar to company model component 118 (shown in FIG. 1and described herein).

In various implementations, method 400 may further comprise training thedocument model and the company model. For example, the document modeland the company model may be trained using a backpropagation algorithm.In some implementations, the document model and the company model may betrained individually. In other implementations, the document model andthe company model may be trained in a single training step (i.e., as asingle “full model”). The single “full model” may also be referred toherein as a credit risk model comprising both the document model and thecompany model. In some implementations, training the document model andthe company model may be performed by a processor component the same asor similar to model training component 120 (shown in FIG. 1 anddescribed herein).

For purposes of explanation, numerous specific details are set forth inorder to provide a thorough understanding of the description. It will beappreciated by those having skill in the art that the implementationsdescribed herein may be practiced without these specific details or withan equivalent arrangement. Accordingly, it is to be understood that thetechnology is not limited to the disclosed implementations, but, on thecontrary, is intended to cover modifications and equivalent arrangementsthat are within the spirit and scope of the appended claims. Forexample, it is to be understood that the present technology contemplatesthat, to the extent possible, one or more features of any implementationcan be combined with one or more features of any other implementation.

In some instances, well-known structures and devices are shown in blockdiagram form in order to avoid unnecessarily obscuring the description.In other instances, functional block diagrams and flow diagrams areshown to represent data and logic flows. The components of blockdiagrams and flow diagrams (e.g., modules, blocks, structures, devices,features, etc.) may be variously combined, separated, removed,reordered, and replaced in a manner other than as expressly describedand depicted herein.

Reference in this specification to “one implementation”, “animplementation”, “some implementations”, “various implementations”,“certain implementations”, “other implementations”, “one series ofimplementations”, or the like means that a particular feature, design,structure, or characteristic described in connection with theimplementation is included in at least one implementation of thedisclosure. The appearances of, for example, the phrase “in oneimplementation” or “in an implementation” in various places in thespecification are not necessarily all referring to the sameimplementation, nor are separate or alternative implementations mutuallyexclusive of other implementations. Moreover, whether or not there isexpress reference to an “implementation” or the like, various featuresare described, which may be variously combined and included in someimplementations, but also variously omitted in other implementations.Similarly, various features are described that may be preferences orrequirements for some implementations, but not other implementations.

The language used herein has been principally selected for readabilityand instructional purposes, and it may not have been selected todelineate or circumscribe the inventive subject matter. Otherimplementations, uses and advantages of the invention will be apparentto those skilled in the art from consideration of the specification andpractice of the invention disclosed herein. The specification should beconsidered exemplary only, and the scope of the invention is accordinglyintended to be limited only by the following claims.

What is claimed is:
 1. A system configured to assess the credit qualityof an entity utilizing a document model and a company model, the systemcomprising: one or more physical computer processors configured bycomputer readable instructions to: obtain a set of documents related toan entity; generate, utilizing a document model, document scores anddocument-representation vectors for the set of documents based on textof the documents, the document scores including a document score foreach of the set of documents and the document-representation vectorsincluding a document-representation vector for each of the set ofdocuments, wherein the document score for a single document indicates alikelihood of an occurrence of one or more future financial events forthe entity based on that single document; aggregate the document scoresand document-representation vectors; create, as an output of thedocument model, a document-model-state vector representing relationshipsidentified within each of the set of documents and across the set ofdocuments; and produce, utilizing a company model, a sequence of defaultprobability scores representing overall likelihoods of the occurrence ofone or more future financial events based on the document-model-statevector.
 2. The system of claim 1, wherein the sequence of defaultprobability scores are produced utilizing a neural network comprisingthe document model and the company model.
 3. The system of claim 1,wherein the one or more processors are further configured to: generate acompany score for the entity based on the sequence of defaultprobability scores, the company score comprising a value between zero(0) and one (1) and representing a default probability for the entity.4. The system of claim 1, wherein the one or more future financialevents comprise one or more of default or bankruptcy.
 5. The system ofclaim 1, wherein to produce the sequence of default probability scores,the one or more processors are further configured to: aggregate thedocument-model-state vector and financial information for the entity,wherein the sequence of default probability scores is determined basedon the aggregated document-model-state vector and financial information.6. The system of claim 1, wherein the one or more processors are furtherconfigured to: generate, utilizing the company model, an internal statevector based on the document-model-state vector, wherein the internalstate vector comprises a first company-model-state vector.
 7. The systemof claim 6, wherein the one or more processors are further configuredto: input, into the company model, a second document-model-state vectorand the first company-model-state vector, wherein the seconddocument-model-state vector is based on aggregated document scores anddocument-representation vectors for a second set of documents; produce,utilizing the company model, a second sequence of default probabilityscores representing overall likelihoods of the occurrence of one or morefuture financial events based on at least the seconddocument-model-state vector; and generate, utilizing the company model,a second internal state vector comprising a second company-model-statevector based on the second document-model-state vector and the firstcompany-model-state vector.
 8. The system of claim 1, wherein the one ormore processors are further configured to: train the document model andthe company model in a single training step using a backpropagationalgorithm.
 9. A method for assessing the credit quality of an entityutilizing a document model and a company model, the method comprising:obtaining a set of documents related to an entity; generating, utilizinga document model, document scores and document-representation vectorsfor the set of documents based on text of the documents, the documentscores including a document score for each of the set of documents andthe document-representation vectors including a document-representationvector for each of the set of documents, wherein the document score fora given document indicates a likelihood of an occurrence of one or morefuture financial events for the entity based on that document;aggregating the document scores and document-representation vectors;creating, as an output of the document model, a document-model-statevector representing relationships identified within each of the set ofdocuments and across the set of documents; and producing, utilizing acompany model, a sequence of default probability scores representingoverall likelihoods of the occurrence of one or more future financialevents based on the document-model-state vector.
 10. The method of claim9, wherein the sequence of default probability scores are producedutilizing a neural network comprising the document model and the companymodel.
 11. The method of claim 9, the method further comprising:generating a company score for the entity based on the sequence ofdefault probability scores, the company score comprising a value betweenzero (0) and one (1) and representing a default probability for theentity.
 12. The method of claim 9, wherein the one or more futurefinancial events comprise one or more of default or bankruptcy.
 13. Themethod of claim 9, wherein producing the sequence of default probabilityscores comprises: aggregating the document-model-state vector andfinancial information for the entity, wherein the sequence of defaultprobability scores is determined based on the aggregateddocument-model-state vector and financial information.
 14. The method ofclaim 9, the method further comprising: generating, utilizing thecompany model, an internal state vector based on thedocument-model-state vector, wherein the internal state vector comprisesa first company-model-state vector.
 15. The method of claim 14, themethod further comprising: inputting a second document-model-statevector and the first company-model-state vector into the company model,wherein the second document-model-state vector is based on aggregateddocument scores and document-representation vectors for a second set ofdocuments; producing, utilizing the company model, a second sequence ofdefault probability scores representing overall likelihoods of theoccurrence of one or more future financial events based on at least thesecond document-model-state vector; and generating, utilizing thecompany model, a second internal state vector comprising a secondcompany-model-state vector based on the second document-model-statevector and the first company-model-state vector.
 16. The method of claim9, the method further comprising: training the document model and thecompany model in a single training step using a backpropagationalgorithm.